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Why use NASA-STD-7009? 
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Numerical models continue to 


• Increase in complexity and capability 

• Significant expertise is required for 
understanding them 

• Need for uncertainty quantification is recognized 
across all disciplines 

• Become more important in decision-making 

• Answer questions that can’t be tested except in 
a virtual environment 


• Code limitations and bounds of applicability must 
be well understood 
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We need clear 
communication between 
modelers and 
customers/users 






Why use NASA-STD-7009 (cont’d)? 


NASA-STD-7009 was 

• Developed after the Columbia 
accident to evaluate engineering 
systems 

• Rich history of use 

• Abundance of documentation 

• Adopted by the Human Research 
Program for biomedical models 

• Encourages the use of best 
practices 

• Adaptation is required to keep it 
relevant 

• Approach demonstrated in many 
biomedical applications 

• Guidance document will be 
available soon 






Modeling and Simulation (M&S) Risk Assessment 



The M&S Risk Assessment 

• Each new use of a numerical model should undergo risk assessment 

• Zone color dictates the need for NASA-STD-7009 


Zone color 

NASA-STD-7009 use 



Yellow 

Typically Required 

Green 

Not required 







Credibility Assessment Matrix: Factor Scores 
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NUMERICAL MODEL SCORES 


TARGET SCORES 


/v 


Credibility Assessment 

Evidence 

Technical Review 

Factor 

Weighted 

Overall 

Sufficiency 


Factors 

Score* 

Weight* 

Threshold* 

Score* 

Threshold* 

Score 

Subfactor Score 

Score 

Threshold 

1 

Verification 

2 

0.20 

3 

2 

3 

2 

0.40 



2 

Validation 

2 

0.25 

2 

2 

3 

2 

0.50 



3 

Input Pedigree 

2 

0.10 

3 

2 

3 

2 

0.20 



4 

Results Uncertainty 

0 

0.10 

2 

0 

3 


0.00 

1.75 

2.54 

5 

Results Robustness 

2 

0.10 

2 

2 

3 

2 

0.20 

6 

Use History 

1 

0.15 

2 

N/A 

N/A 

1 

0.15 



7 

M&S Management 

2 

0.05 

3 

N/A 

N/A 

2 

0.10 



8 

People Qualifications 

4 

0.05 

3 

N/A 

N/A 

4 

0.20 




* Maximum = 4; where 0=insufficient evidence and 4=highest fidelity/ rigor achievable 

# Minimum = 0.05, maximum = 0.25 and sum of all weights must equal 1.0 


Credibility assessment matrix 

• Scores represent both customer/end user and supplier for each new application 

• Is a living assessment that changes as the M&S evolves (for better or worse) 

HRP modifications include 


• High emphasis on technical reviews and 

• Weighting factors appropriate to the type of M&S (deterministic, probabilistic and 
statistical models) 


♦ 
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Credibility Assessment Factors 
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Credibility Factor 

Description 

1 

Verification 

Is the problem solved correctly? Are there bugs in the code? 

2 

Validation 

Does the model prove itself against real-world data? 

3 

Input 

Pedigree 

How much confidence is placed in the data used and the approach 
taken to build the model? How well does the model capture the real- 
world scenario? 

4 

Results 

Uncertainty 

How is error assessed? Is it quantified? How much uncertainty is 
due to demographic/situational variation? Parameter uncertainty? 

5 

Results 

Robustness 

What is the model sensitivity to key parameters? Can it be quantified 
over the region of M&S application? 

6 

Use History 

Has the model been used for decision-making? Was it used in the 
area of application? 

7 

M&S 

Management 

What are the processes/documentation developed during M&S 
planning, development and maintenance? 

8 

People 

Qualifications 

Who is providing the guiding vision? Who is performing the 
implementation? What experience and background do they have? 


Factor scores range from 0 (insufficient evidence) to 4 (highest fidelity/rigor) 6 






Credibility Assessment Matrix: Proposed Weighting Strategy nVsa 


WEIGHT 





Credibility Assessment 

Evidence 

Technical Review 

Factor 

Weighted 

Overall 

Sufficiency 


Factors 

Score* 

Weight* 

Threshold* 

Score* 

Threshold* 

Score 

Subfactor Score 

Score 

Threshold 


Verifies^ 

2 

1 0.20 

3 

2 

3 

2 

0.40 



2 

Validation 

2 

0.25 

2 

2 

3 

2 

0.50 



3 

Input Pedigree 

2 

0.10 

3 

2 

3 

2 

0.20 



4 

Results Uncertainty 


0.10 

2 

0 

3 


0.00 

1 7C 

O r/i 

5 

Results Robustness 

iSk 

0.10 

2 

2 

3 

2 

0.20 

l./D 


6 

Use History 

1 

0.15 

2 

N/A 

N/A 

1 

0.15 



7 

M&S Management 

2 

0.05 

3 

N/A 

N/A 

2 

0.10 



8 

People Qualifications 

4 

0.05 

3 

N/A 

N/A 

4 

0.20 




0.05 < W, < 0.25 


Factor Weight (Proposed) 

Deterministic 

Probabilistic 

1 

Verification 

0.2 

0.075 

2 

Validation 

0.25 

0.15 

3 

Input Pedigree 

0.1 

0.275 

4 

Results Uncertainty 

0.1 

0.2 

5 

Results Robustness 

0.1 

0.15 

6 

Use History 

0.15 

0.15 

7 

M&S Management 

0.05 

0.05 

8 

People Qualifications 

0.05 

0.05 

TOTAL 

1.0 

1.0 


V 


WEIGHTED SUBFACTOR 


Subfactor 

Weight 

Evidence Weighting 

0.7 

Technical Review* 

0.3 

TOTAL 

1.0 


Factor and subfactor weights are 
assigned by the customer 

(Maximum weight is 0.3) 
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Credibility Assessment: Technical Review 
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VASA 


TECHNICAL 
REVIEW 


Credibility Assessment 
Factors 

Evidence 

Technical Review 

Factor 

Score 

Weighted 
Subfactor Score 

Overall 

Score 

Sufficiency 

Threshold 

Score* 

Weight* 

Threshold* 

Score* 

Threshold* 

1 

Verification 




2 

3 

2 

0.40 

1.75 

2.54 

v c v 1 1 no n v/i i 




2 

Validation 

2 

0.2S 

1^1 

2 

3 

2 

0.50 

3 

Input P^^ree 

2 

0.10 

3 

2 

3 

2 

0.20 

5 

Resul* Uncertainty 

0 

0.10 

2 

0 

3 


0.00 

Results Robustness 

2 

0.10 

2 

2 

3 

2 

0.20 

6 

Use History 

1 

0.1S 

2 

N/A 

N/A 

1 

0.15 

7 

M&S Management 

2 

0.05 

3 

N/A 

N/A 

2 

0.10 

8 

People Qualifications 

4 

0.05 

3 

N/A 

N/A 

4 

0.20 


Technical Review also provides input on the required threshold and M&S 
readiness for some of the factors 

Customer specifies the level of technical review that is required for the 
application 


Level 

Technical Review 

4 

Favorable external peer review with 
independent factor evaluation 

3 

Favorable external peer review 

2 

Favorable internal peer review 

1 

Favorable informal internal peer review 

0 

Insufficient evidence 
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Credibility Assessment Matrix: The Spider Plot 


The spider plot is essentially a multidimensional histogram to display the 

model and threshold scores for each factor. 




lUEluil 


Results 

Robustness 


■ Score 

■ Threshold 


Verification 


Validation 


M&S 

Management 


Input Pedigree 


Use History 


Results 
Uncertainty 
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Example of Credibility Scoring -*■' With Factor 
Weighting (NASA HRP Implementation) 


Credibility Assessment 
Factors 

Evidence 

Technical Review 

Factor 

Score 

Weighted 
Subfactor Score 

Overall 

Score 

Sufficiency 

Threshold 

Score* 

Weight* 

Threshold* 

Score* 

Threshold* 

_i 

Verification 

2 

0.20 

3 

2 

3 

2 

0.40 

1.75 

2.54 

_2 

Validation 

2 

0.25 

2 

2 

3 

2 

0.50 

3 

Input Pedigree 

2 

0.10 

3 

2 

3 

2 

0.20 

4 

Results Uncertainty 

0 

0.10 

2 

0 

3 

0 

0.00 

_5 

Results Robustness 

2 

0.10 

2 

2 

3 

2 

0.20 

6 

Use History 

1 

0.15 

2 

M/A 

M/A 

1 

0.15 

7 

M&S Management 

2 

0.05 

3 

M/A 

M/A 

2 

0.10 

8 

People Qualifications 

4 

0.05 

3 

M/A 

M/A 

4 

0.20 


* Maximum =4; where 0=insufficient evidence and 4=highest fidelity/rigor achievable 
+ Minimum = 0.05, maximum = 0.25 and sum of all weights must equal 1.0 


Verification 


People 

Qualificatioi 


M&S 
Management \ 



Subfactors 

Weight 

Evidence 

0.7 

Technical Review 

0.3 


? Input Pedigree 


Use History*' 


Results 
Uncertainty 


Results 

Robustness 


Legend 


CAS Score > Threshold 

Exceeds credibility requirements 


Threshold > CAS Score > (Threshold O.S) 

Ready for use 


(Threshold-0.5) > CAS Score > (Threshold- 1.0) 
Use with caution 


CAS Score < (Threshold-1.0) 

| Use not recommended or to be used with EXTREME CAUTION by subject matter experts only 
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National Aeronautics and Space Administration 

Thank you! 
Questions? 






Backups 



nVsa • 

' r 
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Sufficiency Thresholds 
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Establish Credibility Thresholds nas* 


Level 

Verification 

Validation 

Input Pedigree 

Results 

Uncertainty 

Results 

Robustness 

Use History 

M&S 

Management 

People 

Qualifications 

4 

Numerical 
errors small 
for all 
important 
features. 

Results agree 
with real- 
world data. 

Input data agree 
with real-world 
data. 

Non- 

determimstic 
& numerical 
analysis. 

Sensitivity 
known for 
most 

parameters; 

key 

sensitivities 

identified. 

De facto 
standard. 

Continual 

process 

improvement. 

Extensive 
experience in 
and use of 
recommended 
practices for this 
particular M&S. 

3 

Formal 

numerical 

error 

estimation. 

Results agree 
with 

experimental 
data for 
problems of 
interest. 

Input data agree 
with 

experimental 
data for 
problems of 
interest. 

Non- 

determmistic 

analysis. 

Sensitivity 
know’ll for 
many 

parameters. 

Previous 
predictions 
were later 
validated by 
mission data. 

Predictable 

process. 

Advanced 
degree or 
extensive M&S 
experience, and 
recommended 
practice 
knowledge. 

2 

Unit and 
regression 
resting of 
key features. 

Results agree 
with 

experimental 
data or other 
M&S on unit 
problems. 

Input data 
traceable to 
formal 

documentation. 

Deterministic 
analysis or 
expert 
opinion. 

Sensitivity 
knowm for a 
few’ 

parameters. 

Used before 
for critical 
decisions. 

Established 

process. 

Formal M&S 
training and 
experience, and 
recommended 
practice training. 

1 

Conceptual 

and 

mathematical 

models 

verified. 

Conceptual 

and 

mathematical 
models agree 
with simple 
referents. 

Input data 
traceable to 
informal 
documentation. 

Qualitative 

estimates 

Qualitative 

estimates 

Passes 
simple tests. 

Managed 

process. 

Engineering or 
science degree. 

0 

Insufficient 

evidence. 

Insufficient 

evidence. 

Insufficient 

evidence. 

Insufficient 

evidence. 

Insufficient 

evidence. 

Insufficient 

evidence. 

Insufficient 

evidence. 

Insufficient 

evidence. 


M&S Development 

M&S Operations 

Supporting Evidence 


See NASA-STD-7009 for more info 
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NASA 


fn-sllico 

Analysis 


DAP’s Development and Implementation Process 
for Spaceflight Exercise M&S 


Integration 


r 'k . LfeModeler 




Results - Credibility Summary for Og Simulations ^asa- 


Credibility Assessment 
Factors 

Evidence 

Technical Review 

Factor 

Score 

Weighted 
Subfactor Score 

Overall 

Score 

Sufficiency 

Threshold 

Score* 

Weight* 

Threshold* 

Score* 

Threshold* 

1 

Verification 

1 

0.20 

3 

2 

3 


0.26 

1.35 

2.54 

2 

Validation 

1 

0.25 

2 

2 

3 

1.3 

0.33 

3 

Input Pedigree 

1 

0.10 

3 

2 

3 ! 


0.13 

4 

Results Uncertaintv 

1 

0.10 

2 

1 

3 

1 

0.10 

5 

Results Robustness 

1 

0.10 

2 

2 

3 

1.3 

0.13 

6 

Use History 

1 

0.15 

2 

N/A 

N/A 

1 

0.15 

7 

M&S Management 

2 

0.05 

3 

N/A 

N/A 

2 

0.10 

8 

People Qualifications 

3 

0.05 

3 

N/A 

N/A 

3 

0.15 


Credibility estimated via face value and subject matter expert 
inference from 1g results and knowledge of Og exercise with ARED 


legend 


CAS Score > Threshold 

Exceeds credibility requirements 


Threshold > CAS Score > (Threshold 0.S) 

Ready for use 


(Threshold-0.5) > CAS Score > (Threshold- 1.0) 
Use with caution 


CAS Score < (Threshold-1.0) 

| Use not recommended or to be used with EXTREME CAUTION by subject matter experts only 



Subfactors 

Weight 

Evidence 

0.7 

Technical Review 

0.3 


Unweighted - Model would have a CS = 1! 
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Results - Credibility Summary for 1g Simulations nasa- 


Credibility Assessment 
Factors 

Evidence 

Technical Review 

Factor 

Score 

Weighted 
Subfactor Score 

Overall 

Score 

Sufficiency 

Threshold 

Score* 

Weight* 

Threshold* 

Score* 

Threshold* 

1 

Verification 

2 

0.20 

3 

2 

3 

2 

0.40 

1.80 

2.54 

^2 

Validation 

2 

0.25 

2 

2 

3 

2 

0.50 

3 

Input Pedigree 

2 

0.10 

3 

2 

3 

2 

0.20 

4 

Results Uncertainty 

1 

0.10 

2 

1 

3 

1 

0.10 

_5 

Results Robustness 

2 

0.10 

2 

2 

3 

2 

0.20 

6 

Use History 

1 

0.15 

2 

N/A 

N/A 

1 

0.15 

7 

M&S Management 

2 

0.05 

5 

N/A 

N/A 

2 

0.10 

8 

People Qualifications 

3 

0.05 

3 

N/A 

N/A 

3 

0.15 


* Maximum = 4; where 0=insufficient evidence and 4=highest fidelity/ rigor achievable 


+ Minimum = 0.05, maximum = 0.25 and sum of all weights must equal 1.0 



Legend 


CAS Score > Threshold 

Exceeds credibility requirements 


Threshold > CAS Score > (Threshold O.S) 

Ready for use 


(Threshold-0.5) > CAS Score > (Threshold- 1.0) 
Use with caution 


CAS Score < (Threshold-1.0) 

| Use not recommended or to be used with EXTREME CAUTION by subject matter experts only 


Subfactors 

Weight 

Evidence 

0.7 

Technical Review 

0.3 



Unweighted - Model would have a CS = 1! 
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